Hover-based user-interactions with virtual objects within immersive environments

ABSTRACT

Systems and methods for enabling user-interactions with virtual objects (VOs) included in immersive environments (IEs) are provided. A head-mounted display (HMD) device is communicatively coupled with a hover-sensing (HS) device, via a communication session. The HMD device provides an IE to a wearer by displaying a field-of-view (FOV) that includes a VO. The user executes user-interactions, such as 2D and/or 3D hand gestures, fingertip gestures, multi-fingertip gestures, stylus gestures, hover gestures, and the like. The HS device detects the user-interactions and generates interaction data. The interaction data is provided to the HMD device via the communication session. The HMD device updates the FOV and/or the VO based on the interaction data. A physical overlay that includes a 3D protrusion is coupled with the HS device. The overlay is transparent to the hover-sensing capabilities of the HS device. The protrusion provides tactile feedback to the user for the user-interactions.

BACKGROUND

Advancements in computing-technologies have fueled tremendousdevelopment efforts to deploy immersive environments in variouscomputing-applications, such as simulation, gaming, and entertainmentapplications. For instance, various online gaming platforms haveintegrated virtual-reality (VR) environments, augmented-reality (AR)environments, and/or mixed-reality (MR) environments into gameplay.Deploying such immersive environments enables users to view and perceivecomputer-generated three-dimensional (3D) objects, as if the objectswere actually present within the users' perceived environments. Varioushead-mounted display (HMD) devices, such as VR and/or AR headsets, havebeen developed to deploy immersive environments. Such HMD devicesgenerally provide users a display of a field-of-view (FOV) that includescomputer-generated 3D objects. That is to say, HMD devices provide adisplay of an immersive environment.

However, to enhance the user's experience within the immersiveenvironment, it is advantageous to enable realistic user-interactionswith computer-generated 3D objects displayed via a HMD device. Forinstance, the immersive experience may be enhanced by enabling a user toselect, control, edit, rotate, translate, or otherwise manipulatevarious computer-generated 3D objects within the FOV provided by a HMDdevice. Furthermore, the user may desire to alter various aspects orcharacteristics of the provided FOV. Thus, for a truly immersiveexperience, the ability for the user to provide input to interact withcomputer-generated 3D objects is of tantamount importance. It is forthese concerns and other concerns that the following disclosure isprovided.

SUMMARY

Embodiments described herein provide methods and systems for providingrealistic and intuitive user-interactions with virtual objects (VOs)included in immersive environments (IEs). More particularly, the variousembodiments are directed towards commutatively coupling, via acommunication session, a head-mounted display (HMD) device with aninteraction-sensing (IS) device. The HMD device provides an IE to awearer by displaying a field-of-view (FOV) that includes one or moreVOs.

The user executes one or more gestures (i.e. user-interactions), such asbut not limited to 2D and/or 3D hand gestures, fingertip gestures,multi-fingertip gestures, stylus gestures, hover gestures, and the like.The IS device detects, senses, and/or tracks the user-interactions. Inresponse to such detections of user-interactions, the IS devicegenerates interaction data, and provides at least a portion of theinteraction data to the HMD device via the communication session. TheFOV and/or one or more VOs included in the FOV are updated and/ormodified in response to the interaction data received at the HMD device.Updating and/or modifying the FOV and/or the VO may indicate the user'sintended user-interaction with the VO.

In some of the various embodiments, the IS device is separate from theHMD device, i.e. the IS device is not embedded within and/or notintegrated with the HMD device. The IS device may include at least oneof a hover-sensing (HS) device, a touch-and-hover (TAH) device, or acombination thereof. In other various embodiments, the IS deviceincludes at least a 2D touch-sensitive device. In at least one of thevarious embodiments, the IS device includes multiple camera devices thatdetect and encode, via interaction data, the user-interactions.

In some embodiments, a physical overlay may be coupled with the ISdevice. For instance, when coupled to a HS device and/or a TAH device,one or more surfaces of the overlay (e.g. a protrusion) may be displacedfrom the active surface of the HS device and/or the TAH device. Theoverlay may be constructed from a material that is at least partiallytransparent to the hover-sensing capabilities of the HS and/or the TAHdevice. Thus, the user may touch the displaced surfaces of the overlay,and the hover-sensing capabilities of the HS device and/or the TAHdevice may detect the user's fingertips on the displaced surfaces. Thedisplaced surfaces of overlay may provide one or more 2D and/or 3Dshapes and/or protrusions. Such shapes or protrusions may include, butare not limited to curved bosses, parallelepipeds, cylinders, pyramids,and the like. Thus, the shapes and/or protrusions of displaced surfacesof the overlay provide tactile feedback for the user, when interactingwith VOs.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in detail below with reference to theattached drawing figures, wherein:

FIG. 1 is a block diagram of an exemplary computing environment thatincludes an interaction-sensing device communicatively coupled to ahead-mounted display device and is suitable for use in implementingembodiments of the present disclosure;

FIG. 2A is a schematic showing of an exemplary user-interaction with avirtual object that is enabled via an interaction-sensing device, inaccordance with some embodiments of the present disclosure;

FIG. 2B is a schematic showing of an exemplary embodiment of a userselecting a layer of a virtual object to correlate with atouch-sensitive surface of an interaction-sensing device, in accordancewith some embodiments of the present disclosure;

FIG. 2C is a schematic showing of an exemplary embodiment of a usermodifying the selected layer of the virtual object by touching thetouch-sensitive surface of the interaction sensing-device, in accordancewith some embodiments of the present disclosure;

FIG. 3A is a schematic showing of a step of mechanically coupling anexemplary embodiment of an overlay with an interaction-sensing device,in accordance with some embodiments of the present disclosure;

FIG. 3B is a schematic showing of an exemplary embodiment of a useremploying the mechanically coupled overlay and interaction-sensingdevice of FIG. 3A to update a rotational orientation of a virtualobject, in accordance with some embodiments of the present disclosure;

FIG. 4A is a schematic showing of an exemplary embodiment of a useremploying a interaction-sensing device and another mechanically coupledoverlay to update a position of a virtual object, in accordance withsome embodiments of the present disclosure;

FIG. 4B is a schematic showing of an exemplary embodiment of a useremploying the interaction-sensing device of FIG. 4A and anothermechanically coupled overlay to generate a virtual object within animmersive environment, in accordance with some embodiments of thepresent disclosure;

FIG. 5 is a schematic showing of an exemplary step of mechanicallycoupling an overlay that includes a three-dimensional surface andcapacitive couplers with an interaction-sensing device that includes atwo-dimensional capacitive-sensing surface, in accordance with someembodiments of the present disclosure;

FIG. 6 is a schematic showing of an exemplary embodiment of a useremploying camera system as an interaction-sensing device and ahead-mounted display device to generate a virtual object within animmersive environment;

FIG. 7 is a flow diagram showing of an exemplary embodiment of a methodfor enabling user-interactions with virtual objects, in accordance withsome embodiments of the present disclosure;

FIG. 8 is a flow diagram showing of another exemplary embodiment of amethod for enabling user-interactions with virtual objects, inaccordance with some embodiments of the present disclosure;

FIG. 9 is a block diagram of an exemplary head-mounted display device,in accordance with some embodiments of the present disclosure; and

FIG. 10 is a block diagram of an exemplary computing environmentsuitable for use in implementing embodiments of the present disclosure.

DETAILED DESCRIPTION

As used herein, the term “virtualized object” (VO) may refer to anycomputer-generated object or any computer-generated visualization of a(physical or non-physical) object. Such computer-generated objects (orcomputer-generated visualizations thereof) may be one-dimensional (1D)objects, two-dimensional (2D), or three-dimensional (3D) objects (orvisualizations thereof). As used herein, the term “immersiveenvironment” (IE) may refer to any physical (i.e. “real”) environment,any non-physical (i.e. “computer-generated” and/or “virtual”)environment, or any combination thereof that includes a display of oneor more VOs. Such IEs include, but are not otherwise limited tovirtual-reality (VR) environments, augmented-reality (AR) environments,and/or mixed-reality (MR) environments.

As used herein, the term “interaction data” may refer to any data orinformation that indicates or encodes user-interactions (e.g. hand,fingertip, and multi-fingertip gestures). As used herein, an“interaction-sensing device,” or simply a “IS device” may be any devicethat detects, senses, and/or tracks user-interactions, such as but notlimited to hand gestures, fingertip gestures, multi-fingertip gestures,stylus gestures, and the like. In response to such detections ofuser-interactions, an IS device may generate interaction data thatencodes the user-interactions. The generated interaction data may encode2D and/or 3D hand, fingertip, or multi-fingertip gestures executed by auser. In some embodiments, an IS device may encode gestures of a stylusor a stylus-tip executed by a user.

In various embodiments, a IS device includes one or more wired and/orwireless data transceivers to provide another device, via acommunication session, at least a portion of the generated interactiondata. Such wireless data transceivers include, but are not otherwiselimited to one or more communication radios. In some embodiments, a ISdevice may include a haptic-feedback interface, similar tohaptic-feedback interfaces frequently included in mobile devices such astablets and smartphones.

Development efforts in computing technologies have recently enabled“hover”-sensing (HS) devices. Such HS devices generally detect“hover-gestures” of hands, fingers, and fingertips (i.e. extremities) ofa user, as well as the gestures of a stylus held by a user. Typically, aHS device includes one or more (planar and/or non-planar) activesurfaces and associated hover-gesture sensors. Hover-gesture sensors mayinclude proximity sensors. The associated proximity sensors detect thepresence (or non-presence) of user extremities (e.g. the user's hands,fingers, and fingertips) in proximity to the one or more activesurfaces. For some HS devices, the proximity sensors can determine,sense, and/or detect at approximate least 3D positions or locations ofthe user extremities (relative to the associated active surface), whilethe extremities are hovering and/or in motion proximate the activesurface.

More specifically, when the user extremities are proximate to the one ormore surfaces, a HS device is enabled to generate interaction data thatindicates or encodes the 3D location, position, and/or motion of userextremities relative to the one or more active surfaces. Although thedetection and encoding of the location and/or motion of the user'sextremities are discussed throughout, it should be understood that a HSdevice may be employed to detect and encode other user-interactions,such as those initiated via a stylus held be the user.

The proximity sensors may be implemented via capacitive-sensingtechnologies, similar to those employed in touch-sensitive displaydevices. Typically, a HS device can detect user extremities that arewithin a threshold distance (i.e. a proximate-distance threshold) fromthe active surface. Different HS devices are associated with variousproximate-distance thresholds. For some HS devices, theproximate-distance threshold is between 2.0 cm and 20.0 cm. Theproximate-distance threshold of some HS devices is less than 10.0 cm.For at least one HS device, the proximate-distance threshold is about4.0 cm.

Thus, a HS device may detect (and encode via interaction data)hover-gestures, similar to fingertip and multi-fingertip gestures oftenassociated with touch-sensitive display devices. However, in contrast tothe 2D fingertip gestures associated with touch-sensitive displaydevices, such hover-gestures may be 3D fingertip and multi-fingertipgestures. For instance, for a planar active surface of a HS device, aCartesian coordinate system may be employed. The x-y plane of theCartesian coordinate system may be substantially co-planar with (and thez-axis of the Cartesian coordinate system may be substantiallyorthogonal to) the planar active surface of a HS device.

More particularly, capacitive proximity sensors embedded in and/orintegrated with a planar active surface of a HS device detect the user'sfingertips that are hovering and/or in motions proximate the activesurface. In response to the detection, the HS device generatesinteraction data encoding the x-coordinate, y-coordinate, andz-coordinate of one or more of the user's fingertips (relative to theactive surface), as long as the z-coordinate of fingertip is less thanthe proximate-distance threshold of the proximity sensors associatedwith the active surface. The resolution of the proximity sensors in eachdimension may be similar or may be different, depending on the specificproximity sensors embedded in the HS device. For HS devices withnon-planar active surfaces, other orthonormal 3D coordinate systems maybe employed, such as but not limited to spherical coordinates orcylindrical coordinates, depending on the shape of the non-planar activesurfaces.

As such, a HS device may be employed to detect and encode varioususer-interactions, such as but not limited to 3D-versions of any 2Dgesture associated with touch-sensitive displace devices, such as butnot limited to 3D versions of “pinch,” “pull,” “press and hold,”“single-finger tap,” “two-finger tap,” “single tap,” “double tap,”“swipe left,” “swipe right”, pressure-variance (e.g. 3D-touch) gestures,and the like. Furthermore, a HS device may be employed to detect andencode, hand-writing, notes, sketches, drawings, and otheruser-interactions initiated via the user's fingertips and/or a stylusheld and guided by the user's hands.

The hover-sensing capabilities of a HS device may be integrated with atouch-sensitive device, such as but not limited to a touch-sensitivedisplay device, to generate a touch-and-hover (TAH) device. Thetouch-sensitive device detects and encodes the 2D touch of the user'sextremities, as well as the 3D hover-gestures of the user's extremities.That is, the active surface functions similar to the touch-sensitivesurface of a touch-sensitive display device. As such, a TAH device maydetect and encode all of the 2D fingertip, multi-fingertip, andstylus-tip gestures (on the active surface) associated with atouch-sensitive display device, as well as the 3D fingertip,multi-fingertip, and stylus-tip gestures in proximity to the activesurface. Thus, a TAH device is a HS device. The touch-sensitive surfaceof a TAH device may also be a display device, but need not be.

Thus, in some embodiments, a IS device includes at least one of a HSdevice, a TAH device, or a combination thereof. In other variousembodiments, a IS device includes at least a 2D touch-sensitive device.In some embodiments, the IS device may include a 2D touch-sensitivedisplay device. In other embodiments, the IS device does not include adisplay device. For instance, an IS device may include a 2Dtouch-sensitive tracking pad or a TAH device that does not include adisplay device (i.e. the touch-sensitive device is not a displaydevice). In at least one of the various embodiments, an IS deviceincludes a gaming and/or an entertainment system that includes variouscombinations of proximity sensors, motion sensors, cameras, and thelike. Such combinations of cameras and/or motion sensors may be enabledto detect and encode the user's movements through gameplay, viagenerated interaction data.

In the various embodiments, IS devices may include touch-sensitivedevice, such as a touch-sensitive display device, where thetouch-sensitive device is a multi-touch display device. That is, thetouch-sensitive device that detects and/or senses two or more points ofcontact. In addition to being touch-sensitive, an IS device may includea pressure-sensitive device, such as a pressure-sensitive displaydevice. Such IS device may detect 3D gestures via multi-pressure touchor fingertip gestures. That is a user may provide 3D gestures by thepressure-sensitive device via pressure-variance gestures, e.g.,3D-touch. For instance, the 3D information provided via a HS may besimilarly provided via pressure-variance gestures and apressure-sensitive device.

In some embodiments, a IS device may include any combination of atouch-sensitive, pressure-sensitive device, hover-sensitive device. Thatis, a touch-sensitive device included in a IS device may also bepressure-sensitive and/or hover-sensitive. Thus, a IS device may detect3D via touch, pressure, and/or hovering-like gestures. In otherembodiments herein, 3D gestures may be detected via one or more devices,such as IS devices by employing one or more photon-detectors (e.g., acamera) to track the motion of fingertips hovering over a 2Dtouch-sensitive surface. For instance, a matrix of photon-detectors maybe employed to detect gestures my detecting photons reflected from theusers fingertips. Similarly, an IS device may track the user's gesturesvia acoustic and/or ultra-sound imaging. In some embodiments, an ISdevice may include magnetic sensors and the users wears magnetic rings.The IS device may detect the 3D gestures via the magnetic sensors andthe magnetic rings. In other embodiments, user gestures may be detectedthe sensing of interaction with mechanical and/or active devices orparts, such as but not limited to switches, dials, levers, buttons,joysticks, trackballs, clickable devices, and the like. An IS device mayinclude data gloves and other sensing devices, such as exoskeletons.Such IS devices may be enabled to detect gestures over a mobile device,such as a tablet. Any of these IS devices may also provide hapticfeedback in the various embodiments.

In addition to HS devices and TAH devices, development efforts incomputing technologies have also resulted in the development of varioushead-mounted display (HMD) devices. Such HMD devices generally enablethe realistic perception of computer-generated two-dimensional (2D)and/or three-dimensional (3D) objects. More particularly, an HMD deviceprovides, to a user wearing the HMD device, a visualization (or display)of a field-of-view (FOV) that includes one or more virtualized objects(VOs). As such, an HMD device provides a display or visualization of VOswithin an immersive environment (IE).

A user may wish to interact with the VOs within an IE. For example, auser may wish to select, control, edit, rotate, translate, or otherwisemanipulate a VO. Similarly, a user may wish to interact with variousdiscrete features, structures, components, modules, controls, or otherparts of a VO. Some HMD devices enable limited user-interactions withVOs. For instance, some HMD devices include embedded motion sensors(e.g. cameras) that detect broad user hand-gestures through free space.Upon detection of such a free-space gesture, the HMD device may modify aVO.

However, the spatial-resolution of such embedded motion sensors (andthus the user-interactions) may be limited. For instance, unless theuser positions their hands in close proximity to HMD device, the angularresolution of optical systems associated with the embedded motionsensors may limit the ability to resolve relative smallspatial-differences in the position of the user's hands. Such limitedresolution may limit the user's ability to select, edit, or otherwisemanipulate closely-spaced VOs or closely-spaced features of a VO.Additionally, such HMD devices may require a direct opticalline-of-sight between the user's hands and the embedded motion sensors.For example, if an optically opaque surface (e.g. a desktop) is betweenthe user's hands and the HMD device, or if the user positions theirhands in proximity of their waist, the detection functionalities ofembedded motion sensors may be limited.

In addition to spatial-resolution considerations, hand gestures throughfree space lack tactile and haptic feedback associated with otherphysical objects. Considering an example where a user desires to selectand rotate a virtualized object (e.g. the user wants to select androtate a virtual globe within an IE), a user employing free-space handgestures does not experience the feedback of an associated physicalobject interacting with their hand or fingertips. Additionally, in someimmersive applications, there may be limits to the alloweduser-interactions. For example, within an immersive gaming-environment,there may be virtualized walls or other VOs that the user's hands (or aselected VO) may collide with. Free-space hand-gesture initiateduser-interactions lack tactile and haptic feedback of such collisionevents. Additionally, such unrestrained free-space gestures may notprovide intuitive user-interactions for the manipulations of VOs thatare proxies to physical objects.

Furthermore, at least due to the significant adoption of touch-sensitivedisplay devices, users have become accustomed to interacting withapplications running on computing devices, via intuitive fingertipgestures. For instance, since the integration of touch-sensitive displaydevices within mobile devices (e.g. smartphones, tablets, smartwatches,and the like), users have become familiar with user-interactions withinmobile applications via intuitive fingertip gestures. Such fingertipgestures may include multi-touch (or multi-fingertip) gestures, such asbut not otherwise limited to “pinch,” “pull,” “press and hold,”“single-finger tap,” “two-finger tap,” “single tap,” “double tap,”“swipe left,” “swipe right”, pressure-variance (e.g. 3D-touch) gestures,and the like. Multi-touch gestures may even enable simultaneousinteractions with multiple components of a computing application. Forexample, a user may simultaneously select multiple icons on a smartphonevia multi fingertip touches to separate potions of the touch-sensitivedisplay device.

Such heavily-adopted intuitive user-interactions may be difficult toimplement via hand gestures though free space that are detected viamotion sensors embedded in a HMD device. For instance, it may bedifficult for resolution-limited HMD-device motion sensors to resolvethe spatial resolution of multiple closely-spaced fingertips. That is tosay, the spatial resolution of such a touch-sensitive device may begreater than that of motion sensors. Another issue with conventionmotion sensors is occlusions. For instance, some of the user's fingersmay occlude other fingers of the user. Also, the detection of fingertipgestures that are based on physical contact with a touch-sensitivedisplay device is a challenge. For example, hand gestures through freespace may not realistically emulate the user-experience associated with“tap,” and “press and hold” fingertip gestures that are popular withtouch-sensitive display devices. That is to say, free-space handgestures lack the tactile feedback of tapping and swiping on atouch-sensitive display device.

Other limitations of free space user-interactions include that the usermust suspend their hands in free space without support. The user'smuscle may begin to tire after long usage. Additionally, the length ofthe user's arm is limited. In some situations, the arm of the user maybe used as a pointer, at the expense of the resolution or accuracy ofthe gesture detecting. Such limitations of free space user-interactions,associated with motion sensors embedded in a HMD device, may decreasethe “immersive experience” for the user interacting with VOs within anIE.

The various embodiments herein are directed to systems and methods thatenable realistic and intuitive user-interactions with VOs included in anIE. As discussed throughout, the various embodiments herein address eachof the above noted limitations of previously available methods andsystems for interacting with VOs. More particularly, the variousembodiments are directed towards commutatively coupling, via acommunication session, a head-mounted display (HMD) device with aninteraction-sensing (IS) device. The user executes one or more gestures,such as but not limited to 2D and/or 3D hand gestures, fingertipgestures, multi-fingertip gestures, stylus gestures, and the like. Byexecuting such gestures, the user intends to interact with one or moreof the VOs included in the IE (i.e. the user intends to executeuser-interactions with a VO).

The IS device detects, senses, and/or tracks the user-interactions. Inresponse to such detections of user-interactions, the IS devicegenerates interaction data, and provides at least a portion of theinteraction data to the HMD device via the communication session. TheFOV and/or one or more VOs within the IE are updated and/or modified inresponse to the interaction data received at the HMD device. Updatingand/or modifying the FOV and/or the VO may indicate the user's intendeduser-interaction with the VO.

In some of the various embodiments, the IS device is separate from theHMD device, i.e. the IS device is not embedded within and/or notintegrated with the HMD device. The IS device may include at least oneof a HS device, a TAH device, or a combination thereof. In other variousembodiments, the IS device includes at least a 2D touch-sensitivedevice. In at least one of the various embodiments, the IS deviceincludes multiple camera devices that detect and encode, via interactiondata, the user-interactions.

In some embodiments, a physical overlay may be coupled with the ISdevice. For instance, when coupled to a HS device and/or a TAH device,one or more surfaces of the overlay may be displaced from the activesurface of the HS device and/or the TAH device. The overlay may beconstructed from a material that is at least partially transparent tothe hover-sensing capabilities of the HS and/or the TAH device. Thus,the user may touch the displaced surfaces of the overlay, and thehover-sensing capabilities of the HS device and/or the TAH device maydetect the user's fingertips on the displaced surfaces. The displacedsurfaces of overlay may provide one or more 2D and/or 3D shapes and/orprotrusions. Such shapes or protrusions may include, but are not limitedto curved bosses, parallelepipeds, cylinders, pyramids, and the like.Thus, the shapes and/or protrusions of displaced surfaces of the overlayprovide tactile feedback for the user, when interacting with VOs.

In various embodiments, the IS device may automatically determine anidentifier associated with the overlay. A mode of the MHD device may beupdated based on the identifier of the overlay. In some embodiments, themotion of the user's fingertips along the displaced surfaces of theoverlay may be detected and encoded via the interaction data. Theinteraction data is employed to determine the shape and/or identify theprotrusions provided by the displaced surfaces. The FOV and/or a VO maybe updated and/or modified based on the determined shape.

In various embodiment, the 2D surface of the IS may be mapped to the 2Dsurface of a 3D virtual or real object in the environment. Moving afinger over the touch surface, will move a point on the surface of theobject, where the mapping between the surface area and the IS surface isdefine the same way, as definition of texture mapping. Furthermore,hovering or elevating a finger above the IS surface, will move thecorresponding 3D point, in the direction of the 3D surface normal, awayfrom the object.

A haptic-feedback interface included in the IS device may be employed toprovide haptic feedback to the user in accordance with events occurringwithin the FOV. Thus, the shapes and/or protrusions of an overlay, aswell as the haptic-feedback interface of the IS device provide the usertactile and/or haptic feedback when interaction with VOs. When anoverlay is not coupled with the IS device, the user may additionallyinteract with a VO, via 2D fingertip and multi-fingertip gestures on thetouch-sensitive device of the IS device. Thus, the user may be providedwith tactile feedback via the touch-sensitive surfaces of a IS device.Furthermore, as discussed throughout, hover gestures detected via thevarious IS devices may enable more precise and accurateuser-interactions with VOs, given the increased spatial resolutions ofhover-sensing and touch-sensing active surfaces of IS devices.Additionally, surfaces of an IS, such as but not limited totouch-sensitive surfaces and surfaces of an IS overlay provide supportand friction for the user's fingertips. Such friction may enable precisemovement of the user's fingers.

With reference now to FIG. 1, a block diagram of an exemplary computingenvironment 100 that is suitable for use in implementing embodiments ofthe present disclosure. System 100 includes an interaction-sensing (IS)device 110 and a head-mounted display (HMD) device 140. Moreparticularly, FIG. 1 shows both a non-limiting physical form factorembodiment, and a non-limiting component block diagram, for each of ISdevice 110 and HMD device 140.

A communication network 160 communicatively couples IS device 110 andHMD device 140, via a communication session. In some embodiments,communication network 160 is peer-to-peer (P2P) network that enables thecommunication session between IS device 110 and HMD device 140. Forinstance, IS device 110 and HMD device 140 may be paired via a P2Pcommunication session. In other embodiments, communication network 160may include, without limitation, one or more local area networks (LANs)and/or wide area networks (WANs). In exemplary implementations, network160 comprises the Internet and/or a cellular network, amongst any of avariety of possible public and/or private networks.

Although FIG. 1 shows a single IS device coupled to HMD device 140, itshould be notes that multiple IS devices could be coupled to HMD device140. For instance, at least two IS devices, one for each hand of a user,may be coupled to HMD device 140. In other embodiments, one or more ISdevices may be coupled to HMD device 140, where one or more users of theone or more coupled IS devices are not wearing the HMD device 140. Thatis, a user of IS device 110 may not be the same user that is wearing HMDdevice 140. Similarly, IS device 110 may be couple to more than one HMDdevice. That is, a single IS device may be coupled and/or paired withmultiple HMD devices.

IS device 110 may include a 2D touch-sensitive surface 112.Touch-sensitive surface 112 may include a touch-sensitive device, suchas but not limited to a touch-sensitive display device similar to thosefrequently used in computing devices, such as smartphones and tablets.However, touch-sensitive surface 112 need not be a display device. Asshown in FIG. 1. Touch-sensitive surface 112 may be a planar surface,but may alternatively include non-planar (i.e. curved) surfaces.

IS device 110 includes touch-gesture sensors 120 that sense or detecttouch-gestures on touch-sensitive surface 112. More particularly,touch-gesture sensors 120 may sense and/or detect 2D touch-gesturesassociated with touch-sensitive devices. That us, touch-gesture sensors120 may sense and/or detect hand, fingertip, multi-fingertip, and stylusgestures such as but not limited to “pinch,” “pull,” “press and hold,”“single-finger tap,” “two-finger tap,” “single tap,” “double tap,”“swipe left,” “swipe right”, pressure-variance (e.g. 3D-touch) gestures,and the like. In various embodiments, touch-gesture sensors 120 may becaptivate-sensing sensors. Thus, touch-sensitive surface 112 may be a 2Dcapacitive-sensing surface.

In some embodiments, IS device 110 may be a hover-sensing (HS) device.In at least one embodiment, IS device 110 may be a touch-and-hover (TAH)device. In such embodiments, touch-sensitive surface 112 mayadditionally be an active surface of the HS device, i.e. IS device 110may sense hover gestures in proximity to touch-sensitive surface 112.Accordingly, surface 112 may be both touch-sensitive andhover-sensitive. More particularly, IS device may include hover-gesturesensors 122, such as but not limited to proximity and/or motion sensors,that sense and/or detect hover-gestures in proximity to touch-sensitivesurface 112. Such hover gestures include but are not limited to 3D hand,fingertip, multi-fingertip, and stylus gestures. For instance,hover-gesture sensors 122 may sense and/or detect 3D versions offingertip gestures associated with touch-sensitive devices, such as butnot limited to 3D versions of “pinch,” “pull,” “press and hold,”“single-finger tap,” “two-finger tap,” “single tap,” “double tap,”“swipe left,” “swipe right”, pressure-variance (e.g. 3D-touch) gestures,and the like.

In various embodiments, hover-gesture sensors 122 may becaptivate-sensing sensors. Thus, touch-sensitive surface 112 may be a 3Dcapacitive-sensing surface. In at least one embodiment, IS device 110does not include a touch-sensitive surface and/or touch-sensitivesensors. That is, surface 112 may be only a hover-sensing surface (i.e.surface 112 is not a touch-sensitive surface).

IS device 110 includes an interaction-data generator 124 that generatesinteraction data that encodes the touch and hover gestures sensed and/ordetected via touch-gesture sensors 120 and hover-gesture sensors 122respectively. Interaction-data generator 124 may process, package,encrypt, or other otherwise prepare the interaction data fortransmission to HMD device 140. IS device 110 includes ISdata-transceiver 128, which may be a wired or a wireless datatransceiver. Such wireless data transceivers include, but are nototherwise limited to one or more communication radios. ISdata-transceiver 128 is enabled to provide HMD device 140 theinteraction data, via the communication session enabled by communicationnetwork 160.

In some embodiments, IS device 110 includes a haptic-feedback interface,such as those commonly integrated in smartphones, tablets, video-gamecontroller devices, and the like. IS device 110 may include a pluralityof camera devices that sense and/or detect free-space gestures of auser. Interaction-data generator 124 may generate interaction dataencoding such free-space gestures. In various embodiments, IS device 110includes one or more additional components included in a computingdevice. In at least one embodiment, IS device 110 device includes a IScomputing device 130. Various embodiments of computing devices arediscussed in conjunction with at least FIG. 10.

HMD device 140 includes an immersive-environment (IE) display device 142that is enabled to provide a display and/or a visualization of one ormore virtual objects (VOs) to a user wearing HMD device 140. HMD device140 also includes a wired and/or wireless data transceiver, such as HMDdata-transceiver 114. HMD data-transceiver is enabled to receive theinteraction data, provided via IS data-transceiver 128 of IS device 110.MHD device 140 may additionally include a computing device, such as HMDcomputing device 146. Other embodiments of a HMD device are discussed inconjunction with at least FIG. 9.

FIG. 2A is a schematic 200 showing an exemplary user-interaction with avirtual object (VO) that is enabled via interaction-sensing (IS) device210, in accordance with some embodiments of the present disclosure. ISdevice 210 may include similar features to those of IS device 110 ofFIG. 1. As such, IS device 110 may be a HS device and/or a TAH device.IS device 110 includes a surface 212. Surface 212 may be a hover-sensingsurface, a touch-sensitive surface, and/or a combination thereof. ACartesian coordinate system is shown in FIG. 1. The x-y plane of theCartesian coordinate system is substantially coincident with the planarsurface 212. The z-axis of the coordinate system is substantiallyorthogonal to planar surface 212.

IS device 110 may be paired with a HMD device, such as but not limitedto HMD device 140 of FIG. 1, via a communication session. The HMD deviceprovides a field-of-view (FOV) 250 to a user wearing the HMD device. FOV250 includes various virtualized objects (VOs) that represent a 3D viewof the solar system. It should be understood that other FOVs and VOs areconsistent with the various embodiments. One VO included in FOV 250 isselection cursor 252. Other VOs included in FOV 250 includes, but arenot limited to holograms or 3D visualizations of Earth 256, the Sun 254,and Saturn 258.

IS device 210 may generate interaction data that encodes the touch andhover gestures of a user. The interaction data may be provided to theHMD device, via the communication session. The HMD device may updateand/or modify the FOV 250 and/or any VOs included in FOV 250 based onthe interaction data. Thus, the user can interact with the FOV 250and/or any VOs included in the FOV 250. For instance, the user mayselect, control, edit, rotate, translate, or otherwise manipulate a VO(or features of a VO). Similarly, a user may alter various aspects,characteristics, or properties the provided FOV 250.

More particularly, FIG. 1 shows the user's hand 202. Multiple fingertips204 of hand 202 are shown in a 3D hover-gesture. Note, the z-coordinateof each of the fingertips 204 is greater than 0.0, i.e. the fingertips204 are “hovering” above (and not touching) hover-sensing surface 212.The 3D hover gestures of a user may control the operation of selectioncursor 252 within the 3D FOV 250. For instance, the user may manipulatethe 3D location or position of the selection cursor 252 within the 3DFOV 250. The hover-gestures may be employed to select, control, orotherwise manipulate other VOs included in FOV 250, such as but notlimited to the holograms or visualizations of the Sun 254, the Earth256, or Saturn 258.

Thus, virtually any user-interaction with a VO within an IE may beenabled via the communicatively coupling of IS device 210 with a HMDdevice. The spatial-resolution of the touch and hover gestures encodedin the interaction data may be greater than the spatial-resolutionassociated with detecting free-space gestures via motion sensorsembedded within the HMD device. Accordingly, the spatial-resolutionassociated with user-interactions enabled via IS device 210 may begreater than the spatial-resolution associated with previously availablesystems and methods for enabling user-interactions with VOs.Additionally, IS device 210 enables interacting with VOs within an IEvia 2D and 3D versions of pinch,” “pull,” “press and hold,”“single-finger tap,” “two-finger tap,” “single tap,” “double tap,”“swipe left,” “swipe right”, pressure-variance (e.g. 3D-touch) gestures,and other fingertip and multi-fingertip gestures commonly employed byusers of 2D touch-sensitive display devices. The emulation of such 3Duser-interactions may not be as readily accomplished via previouslyavailable methods of detecting free-space gestures.

In the various embodiments, touch and hover gestures may be combined togenerate additional gestures. For instance, one finger may touch thedisplay for support, while another finger is hovering to define positionin space. In such embodiments, the hand is supported, as well as theuser muscles are used to sense the distance from the IS in a veryaccurate and non-visual way. In at least one embodiment, the distancebetween the finger on the IS device and the hovering finger, can definea vertical scale of an object. The position of a second finger (such asthe thumb) can change the meaning of the touch gesture (for example fromdragging to selection).

Although FIG. 2A shows a single IS device 210 paired with an HMD device,it should be understood that multiple IS devices may be paired with aMHD device. For instance, at least two IS devices may be paired with anHMD device, one for each hand of the user. Accordingly, the user may seteach IS device on a stable surface, such as a table, and may controlmultiple VOs via multiple hand gestures, where each hand hovers over arespective IS device. Additionally, one or more IS devices may becoupled to the HMD device, where the users of the one or more IS devicesare separate and/or remote from the user that is wearing the HMD device.Similarly, a single IS device may be coupled to multiple HMD devices.For instance, a user of IS device 210 may control and/or manipulate oneor more VOs that are simultaneously being shown to multiple users viamultiple HMD devices.

FIG. 2B is a schematic showing of an exemplary embodiment of a userselecting a layer (i.e. a planar slice) of a virtual object to correlatewith a touch-sensitive surface of an interaction-sensing device, inaccordance with some embodiments of the present disclosure. Moreparticularly, FIG. 2 shows a FOV 260 (provided via a HMD device). FOV260 may include either a “flat” or 3D representation 268 of a pluralityof layers 264 of a layered object. For instance, representation 268 maybe a representation of the various layers included in a layereddocument, such as but not limited to a presentation slide.

In the non-limiting embodiment shown in FIG. 2B, each of layers 1-5 of apresentation slide are shown via a “flat” presentation of the pluralityof layers 264, projected and/or displayed within FOV 260. Each layerincludes one or more 2D and/or 3D VOs. Layer 1 is a background layerthat includes a background color or pattern, Layer 2 includes a single(2D or 3D) triangular VO, and Layer 3 includes a (2D or 3D)parallelogram-like VO 272, a (2D or 3D) circular VO 274, and a firsttext box 276 (i.e. “Text_1”). Layer 4 includes two text boxes (i.e.“Text_2” and “Text_3”). Layer 5 includes a 2D or a 3D visualization 266of a solar system, such as the solar system provided in FOV 250 of FIG.2A, that is embedded in Layer 5 of the layered document.

Note, that the presentation of the layers of a multilayered objectwithin a 3D FOV need not be “flat” as depicted in FIG. 2B. That is, themulti-layers may be shown in a more 3D presentation within a FOV.Whether the presentation of the layers is “flat” or 3D, a user mayselect a layer via a selection cursor 270, or some other mechanism. Moreparticularly, the user may employ 3D hover gestures, detected via a ISdevice, such as but not limited to IS device 210 of FIG. 2A to select alayer (or planar slice) of a VO.

The selection of the layer or planar slice of the VO, via a hovergesture detected by the IS device, may result in a “pinning” (orcorrelation) of the layer with a touch-sensitive surface of the ISdevice. That is to say, the plane of the selected layer is pinned orcorrelated with the plane of a touch-sensitive surface of the IS device.More specifically, once a layer or planar slice of a VO (e.g. Layer 3 ofthe presentation slide of FIG. 2B) has been selected, the FOV providedby the HMD device may be transitioned to include a “flat” or 3Dvisualization of the selected layer. Furthermore, once correlated withthe touch-sensitive surface, the user may modify, edit, update, orotherwise manipulate the selected layer (or planar slice) of the VO viafinger or multi-finger 2D touch gestures on the touch-sensitive surface.The user may observe the user-interactions with the VOs within theselected layer via modifications and/or updates within the FOV providedby the HMD device.

A hovering finger may select a plane above the IS device in a continuesmotion upward-downward. The selection of such a plane, may bring acut-out of a 3D object presented above the IS device, to be displayed onthe screen of the IS. In a similar manner, a hovering finger may move a3D virtual object into and out of the FOV of the hover device of the IS.As the object is pushed into FOV of the IS, the IS may display differentlayers/cutouts of the object. In another embodiment the IS device can behold in 3D space inside a 3D object, and by touching the IS, it may copya slice of the 3D object onto the IS display. The user may hold,examine, annotate, or manipulated the object or slice of the object.

FIG. 2C is a schematic showing of an exemplary embodiment of a usermodifying the selected layer of the virtual object by touching thetouch-sensitive surface of the interaction sensing-device, in accordancewith some embodiments of the present disclosure. As shown in FIG. 2C, avisualization of the selected Layer 3 of the presentation slide of FIG.2B is included in FOV 280, via the HMD device. That is to say, FOV 200includes parallelogram-like VO 272, circular VO 274, and text box 276.The user may modify, edit, update, or otherwise manipulate any of theseVOs via finger or multi-finger 2D touch gestures. FIG. 2C shows user'shand 202 and fingertips 204 perform 2D touch gestures withtouch-sensitive surface 212 of IS device 210.

Fingers 204 are in physical contact with touch-sensitive surface 212.More specifically, via common 2D fingertip and multi-fingertip gestures(e.g. “pinch,” “pull,” “press and hold,” “single-finger tap,”“two-finger tap,” “single tap,” “double tap,” “swipe left,” “swiperight”, pressure-variance (e.g. 3D-touch) gestures, and the like), theuser may re-size, rotate, translate, re-position, edit text, orotherwise interact with and manipulate any of VOs 272, 272, and 276.

The touch of the touch-sensitive surface on the user's fingertips mayprovide tactile feedback for the user, enabling more realistic,controlled, and precise user-interactions with various VOs included inthe selected and pinned (or correlated) layer. Furthermore, the spatialresolution of the 2D touch-sensitive surface (i.e. the spatialresolution in the x-y plane) may be greater than the spatial resolutionassociated with 3D hover gestures. That is, the interaction data maymore accurately encode the (x-axis and y-axis) positions and motions ofthe user's fingertips, when the fingertips are in contact with thetouch-sensitive surface, as compared to when the fingertips are hoveringabove the touch-sensitive surface. Thus, by correlating a layer orplanar-slice of a VO object to a touch-sensitive surface of the ISdevice, a user may more precisely and accurately interact with variousVOs within an IE.

FIG. 3A is a schematic showing a step of mechanically coupling anexemplary embodiment of an overlay with an interaction-sensing (IS)device, in accordance with some embodiments of the present disclosure.System 300 includes IS device 310 and overlay 320. Overlay 320 may be aphysical overlay. IS device 310 may be similar to IS device 110 of FIG.1 or IS device 210 of FIGS. 2A-2C. As such, IS device 310 may include anactive surface 312, such that IS device detects hover gestures inproximity to active surface 312 (i.e. hover gestures that are within theproximity-distance threshold of active surface 312).

Overlay 320 is configured and arranged for mechanically coupling orotherwise interfacing with IS device 310. For instance, overlay 320 maybe configured and arranged to “snap” onto or otherwise become fixablyattached to IS device 310. Overlay 320 can include a 3D protrusion 322.In the non-limiting embodiment shown in FIG. 3A, the protrusion 322includes a hemispherical boss shape. In other embodiments, a protrusionof overlay 320 may include virtually any shape. When mechanicallycoupled to IS device 310, the surfaces of protrusion 322 may bedisplaced from the active surface 312. Note however that the displacedsurfaces of protrusion 322 are still within the proximate-distancethreshold of active surface 312 when mechanically coupled to and/orinterfacing with IS device 310.

Overlay 320 may be constructed from plastic, a glass, or anothermaterial that is at least partially transparent to the hover-sensingabilities of IS device 310. For instance, overlay 320 may be constructedvia molded plastic or a tempered glass. Plastics may include, but arenot limited to a polyethylene (PET) plastic or a thermoplasticpolyreuthane (TPU) plastic. In at least one embodiments, overlay 320 mayinclude capacitive sensors to detect and/or sense the user's touch ofsurfaces of overlay 320. In such embodiments, overlay may generate andprovide interaction to IS device 320 via a wired and/or wireless datatransceiver interface. Thus, overlay 320 may not be required to betransparent to the hover-sensing abilities of IS device 310.

As such, IS device 310 is enabled to detect (and generate interactiondata in response to the detection of) when the user touches or positionsone or more fingertips on the surfaces of the overlay 320, such as butnot limited to the displaced surfaces (relative to active surface 312)of protrusion 322. As such, IS device 310 may sense the hover gesturesof a user, while the surfaces of overlay 320, such as but not limited tothe surfaces of protrusion 322, provide the user tactile feedback foruser-interactions with a VO. That is, the user may experience tactilefeedback while interacting with a VO via 3D hover gestures detected withIS device 310.

FIG. 3B is a schematic showing of an exemplary embodiment of a useremploying the mechanically coupled overlay 320 and IS device 310 of FIG.3A to update a rotational orientation of VO 360, in accordance with someembodiments of the present disclosure. More particularly, FIG. 3B showsuser's hand 302 performing hover gestures via fingertips 304 touchingthe displaced surfaces of hemispherical boss-shaped protrusion 322. FIG.3B also shows FOV 350 provided via a HMD device.

FOV 350 includes a VO 360, which is a hologram of Earth. The user mayinteract with VO 360 (or any other VO) via 3D hover gestures withfingertips 304 in contract with surfaces of protrusion 322. Forinstance, the user may manipulate, modify, or update a rotationalorientation of VO 360 by gliding fingertips 304 or the palm of hand 302over the hemispherical surfaces of protrusion 322. In anotherembodiment, the user may rotate a viewpoint of FOV 350 (i.e. rotate aposition of viewer within the IE) based on hover gestures with theirfingers in contact with the protrusion 322. In still other embodiments,the user may employ the hemispherical surfaces of protrusion 322 as a“trackball”-like control element for the interactions with VO 360. Thus,when interacting with VO 360, the user experiences tactile feedback viathe physical sensation of fingertips 304 in physical contract withsurfaces of protrusion 322.

In various embodiments, at least a portion of the surfaces of theoverlay may be constructed from a softer material, such as a pliable ormalleable material that the user may at least slightly deform viapressure from there touch. For instance, the upper portion of protrusion322 may be constructed from such a softer material, while the lowerportion is constructed from a harder or less pliable material, e.g., theupper portion may be overlaid with a putty or soft rubber-type material.The contrast in the hardness of the materials may provide additionaltactile feedback, variances in friction, or otherwise constrain themovement of the fingers.

The constraints may be defined via furrows in the hard and/or softmaterials. In at least one embodiment, at least a portion of theprotrusion may be deformable, e.g., the protrusion may include aninflatable portion or otherwise be deformable via a mechanical devices.As discussed herein, haptic feedback may be selectively provided via theharder and softer surfaces. For instance, an overlay may include one ormore haptic feedback interface. The haptic feedback interface associatedwith the softer surfaces may be activated and/or operated separatelyfrom the haptic feedback interfaces of the harder surfaces.

Although not shown in FIG. 3B, overlay 320 may include mechanical,(e.g., movable parts), and/or protrusions. Overlay 320 may include oneor more active parts, such as but not limited to switches, dials,levers, buttons, joysticks, trackballs, clickable devices, and the like.User interaction with such active devices may enable the generation ofsignals that are employed to control and/or manipulate one or more VOS,such as VO 36 of FIG. 3B.

Although FIG. 3B shows a single overlay 320 couple to IS device 310, itshould be noted that multiple overlays may be coupled to IS device 310.For instance, a first coupled overlay may cover a first portion of theactive surface 312 of IS device and a second coupled overlay may cover asecond portion of the active surface 312. It should be understood thatthere is not an upper bound in the number of overlays that may becoupled to IS device 110. Each of the overlays separate protrusions.That is, each of the multiple overlays may include differently shapedprotrusions.

FIG. 4A is a schematic showing of an exemplary embodiment of a useremploying IS device 410 and another mechanically coupled overlay 420 toupdate a position of VO 460, in accordance with some embodiments of thepresent disclosure. In contrast to the hemispherical boss-shapedprotrusion 322 of overlay 320 of FIGS. 3A-3B, overlay 420 of FIG. 4Aincludes a parallelepiped-shaped protrusion 422. Similar to protrusion322, the displaced surfaces of protrusion 422 are within theproximate-distance threshold of an actual surface of IS device 410.Thus, IS device 410 may detect hover gestures when fingertips are incontact with the surfaces of overlay 420, including the displacedsurfaces of protrusion 422. FIG. 4A shows the user executing hovergestures via fingertips 404 of user's hand 402 being within physicalcontact with and gliding over the displaced surfaces of protrusion 422.

FIG. 4A also shows FOV 450, provided via a HMD device communicativelycoupled to IS device 410. FOV 450 includes VO 460, which is a hologramof Earth. By gliding fingertips 404 over protrusion 422, the user mayperform hover gestures to interact with VO 460. Furthermore, thesurfaces of protrusion 422 provides tactile feedback to the user. Forinstance, the user may translate, update, modify, or otherwisemanipulate the position or location of VO 360 within FOV 450. Thespatial direction of the translation may be based on the particularplanar surface of the parallelepiped-shaped protrusion 422 the userglides fingertips 404 over. For instance, some of the surfaces ofprotrusion 422 lie within the x-y place, other surfaces of protrusionlie within the x-z plane, and other surfaces lie within the y-z plane.Thus, different shapes of a protrusion included in an overlay canprovide different user-interactions with various VOs.

FIG. 4B is a schematic showing of an exemplary embodiment of a useremploying the interaction-sensing device of FIG. 4A and anothermechanically coupled overlay to generate a virtual object within animmersive environment, in accordance with some embodiments of thepresent disclosure. Overlay 430 has been mechanically couple orinterfaces with ID device 410. Overlay includes a pyramid-shapedprotrusion 432. Fingertips 403 of user's hand 402 is shown gliding overthe surfaces of pyramid-shaped protrusion 432. In response to such hovergestures, IS device generates interaction data encoding the positionsand/or motion of fingertips 402 as fingertips glide over the surfaces ofprotrusion 432.

FIG. 4B also shows FOV 480, provided via a HMD device communicativelycoupled to IS device 410. In some embodiments, a shape of protrusion 432may automatically be determined based on the interaction data. Forinstance, either the IS device or the HMD device may determine the shape(i.e. a pyramid) of protrusion 432 based on the encoded positions andmotion of fingertips 402 gliding over protrusion 432. It should be notedthat embodiments are not limited to determining the shape of aprotrusion included in an overlay. That is, a user could hold discrete(i.e. not integrated with overlay 43) physical object (such as but notlimited to a ball or a block) over the active surface of IS device 410.For instance, the user may remove overlay 430. The interaction datagenerated by the user manipulating the physical object (i.e. hovergestures), over the hover-sensing active surface of IS device 410, maybe employed to determine the shape of the manipulated physical object.

The HMD device may update FOV 480 based on the determined shape ofprotrusion 432 (or another discrete physical object) and/or theinteraction data that is used to encode the hover gestures of the user.In one non-limiting embodiment, FOV 480 may be updated to include ahologram or other visualization of a VO of the determined shape. Forexample, FOV 480 has been updated and/or modified to include VO 470,based on the determined pyramid shape of protrusion 432.

Furthermore, a unique identifier associated with overlay 430 mayautomatically be determined based on the determined shape of protrusion432. In other embodiments, IS device 410 may be enabled to automaticallydetermine and/or detect the unique identifier associated with overlay430. For instance, IS device may include various sensors, such as butnot limited to optical scanners, cameras, touch-sensitive sensors,radios, and the like that may automatically detect the mechanicallycoupling of overlay 430 with IS device 410. Such sensors may detect andoptically scannable code, a radio-frequency identification (RFID) tag,or the like that is uniquely associated with the unique identifierassociated with overlay 430.

In various embodiments, the unique identifier may be encoded in theinteraction data provided to the HMD device. An operating mode of theHMD device may be updated based on the identifier. For instance, thevarious user-interactions provided to a user may be updated based on theidentifier associated with the overlay. In other embodiments, theoperating mode may be updated based on the automatically determinedshape of the protrusion or the discrete physical object.

FIG. 5 is a schematic showing of an exemplary step of mechanicallycoupling an overlay 520 that includes a 3D surface (i.e. protrusion 530)and a plurality of capacitive couplers 530 with an interaction-sensingdevice 510 that includes a 2D capacitive-sensing surface (i.e.touch-sensitive surface 512), in accordance with some embodiments of thepresent disclosure. In some embodiments, an IS device, such as but notlimited to IS device 510, do not include an active surface that senses3D hover gestures. Rather, IS device 510 includes a 2D touch-sensitivesurface 512, that may be similar to the touch-sensitive display devicescommonly employed in various mobile computing devices.

When overlay 520 is mechanically coupled to IS device 510, the pluralityof capacitive couplers 530 can capacitively couple at least a portion ofthe points of the 3D surface of protrusion 530 with points on the 2Dtouch-sensitive surface 512. Thus, based on the 3D shape of protrusion522 and the locations (on each of the protrusion 522 and touch-sensitivesurface 512) of the capacitive couplers 530 and the locations of thecorresponding points on the 2D touch-sensitive surface 512, a one-to-onecorrespondence map may be generated. That is, a one-to-one map betweenthe points on the 3D surface of protrusion 522 and the correspondingpoints on the 2D touch-sensitive surface 512 may be generated. In someembodiments, the IS device is enabled to generate the one-to-one map. Inat least one embodiments, the HMD device generates the one-to-one map.

Capacitive couplers 530 and the one-to-one correspondence map can enablethe user performing 3D hover gestures along the surfaces of overlay 510to 3D interact with a VO, employing an IS device that does not includehover-sensing capabilities. For example, a spherical overlay, may beused to rotate a 3D object in VR/AR while looking at it, and feeling thefinger movement on the plastic protrusion.

FIG. 6 is a schematic showing of an exemplary embodiment of a useremploying camera system 610 as an IS device and a HMD device 690 togenerate a virtual object within an immersive environment (IE). Moreparticularly, camera system 610, within room 600 may include one or morecamera devices that track, sense, or detect the free-space gestures ofthe user's hand 602. For instance, camera system 610 may be a componentof a video gaming console and/or entertainment system. Camera system 610may generate interaction data encoding the free-space gestures of user'shand 602 and/or fingertips.

Camera system 610 may be communicatively coupled, via a communicationsession, to HMD device 690. HMD device 690 may be an AR and/or a MRenabled HMD device, such as those discussed in conjunction with HMDdevice 140 of FIG. 1 and/or HMD device 902 of FIG. 9. In response to thereceived interaction data, the HMD device 610 update and/or modify thedisplayed FOV. For instance, HMD device 610 may generate a VO (e.g. therectangular object 660) within the FOV.

Referring now to FIG. 7 in light of FIGS. 1-6, FIG. 7 is a flow diagramshowing of an exemplary embodiment of a method for enablinguser-interactions with virtual objects, in accordance with someembodiments of the present disclosure. Each block of method 700 andother methods and/or processes described herein comprises a computingprocess that may be performed using any combination of hardware,firmware, and/or software. For instance, various functions may becarried out by a processor executing instructions stored in memory. Themethods may also be embodied as computer-usable instructions stored oncomputer storage media. The methods may be provided by a standaloneapplication, a service or hosted service (standalone or in combinationwith another hosted service), or a plug-in to another product, to name afew.

Initially, at block 702, a communication session is established betweenan IS device and a HMD device. In some embodiments, the HMD device isenabled to initiate and establish the communication session. In otherembodiments, the IS device initiates and establishes the communicationsession. In some non-limiting embodiments, the communication session isa peer-to-peer (P2P) communication session, but need not be. At anyrate, via block 702, the IS device and the HMD device arecommunicatively coupled and/or are in communication.

The IS device may be a hover-sensing (HS) device, a touch and hoverdevice, a touch sensing device, a camera system, or any of the othervarious embodiments of IS devices discussed herein. The HMD device maybe a AR HMD device, a VR HMD device, a MR HMD device, or the like.

At block 704, the IS device detects one or more user gestures. Forinstance, the IS device may detect 2D and/or 3D gestures such as hovergestures, free-space hand gestures, fingertip gestures, multi-fingertipgestures, touch gestures, and the like. At block 706, and in response tothe detected user gestures, the IS device generates interaction dataencoding the user gestures.

At block 708, at least a portion of the interaction data is received ator by the HMD device. In some embodiments, the interaction data isreceived directly from the other IS device, via the communicationsession. That is, the IS device provides and/or communicates theinteraction data to the HMD device via the communication session. Inother embodiments, the interaction data is received via anothercomputing device that is communicatively intermediate the IS device andthe HMD device.

At block 710, a VO included in a FOV provided by the HMD device ismodified and/or updated based on the generated interaction data. Themodification to the VO may include virtually any modification to providea visualization and/or other indication of the user interacting with theVO. For instance, the user may select, control, edit, rotate, translate,move, reposition or otherwise manipulate the VO. For instance arotational orientation or a position of the VO may be updated based onthe interaction data. In some embodiments, the FOV may be updated and/ormodified to change the color, contrast, or brightness of a background orone or more VOs included in the FOV. In various embodiments, theviewpoint (e.g. a position of the viewer's perspective) of the FOV maybe updated at block 710. In some embodiments, the FOV may be updated toinclude the generation of new VOs and/or the multiplication and/orreproduction of VOs already included in the FOV.

In at least one embodiment, the interaction data encodes a selection ofa planar slice or a layer of a 3D VO. At block 708 or block 710, theplanar slice of the VO may be correlated with a touch-sensitive surfaceof the IS device. For instance, the selected planar slice may be“pinned” to the touch-sensitive surface. The IS device may generateadditional interaction data, in response to the user executing touchgestures on the touch-sensitive device. Upon receiving the additionalinteraction data, the HMD device may modify and/or update the VO basedon the additional interaction data.

FIG. 8 is a flow diagram showing of another exemplary embodiment of amethod 800 for enabling user-interactions with virtual objects, inaccordance with some embodiments of the present disclosure. Initially,at block 802, an overlay is mechanically coupled to an IS device. The ISdevice may be communicatively coupled to a HMD device, as discussed inconjunction with at least method 700 of FIG. 7. One or more surfaces ofthe overlay (e.g. surfaces of a 3D protrusion) may be displaced from ahover sensing active surface of the IS device. The IS device maygenerate interaction data in response to detecting the user touching theone or more displaced surfaces (e.g. a protrusion).

At block 804, an identifier associated with the overlay may beautomatically determined by at least one of the IS device or the HMDdevice. For instance, the identifier may be automatically determined viaan RFID tag, an optically scannable code, object recognition features,or the like. In at least one embodiment, the identifier is automaticallydetermined based on interaction data generated from the user gliding (ortouching) their fingertips over one or more protrusions of the overlay.

At block 806, interaction data may be communicated to the HMD device. Inaddition to user gestures, the interaction data may encode theidentifier of the overlay. An operating mode of the HMD device may beupdated and/or modified based on the identifier. At block 808, a shapeof a physical object is determined based on interaction data. Forinstance, the physical object may be a protrusion of the overlay and/ora discrete physical object that is separate from the overlay. Theinteraction data may be generated based on hover gestures of the usermanipulating the physical object. The determination of the shape may beperformed at the IS device or the HMD device.

At block 810, the FOV provided by the HMD device is updated and/ormodified based on the determined shape. For instance, a VO may begenerated within the FOV, wherein the shape, color, position, rotationalorientation, or the like of the VO is based on the determined shapeand/or generated interaction data. At block 812, an event is detected inthe FOV. For instance, the event may include an event within a gaming orsimulation application (e.g. a collision of a user controlled VO withanother VO included in the IE). At block 814, and in response to thedetected event, haptic feedback is provided at the IS device. Forinstance, a haptic-feedback interface of the IS device may be employedto provide haptic feedback to the user operating the IS device.

Turning to FIG. 9, a mixed-reality (MR) and/or augmented-reality (AR)HMD device 902 for augmented reality (AR) and MR applications having,among other things, a virtual object rendering component 904, a HMD datatransceiver component 906, and an HMD computing device component 908, isdescribed in accordance with an embodiment described herein. The HMDdevice 902 includes a see-through lens 910 which is placed in front of auser's eye 912, similar to an eyeglass lens. It is contemplated that apair of see-through lenses 910 can be provided, one for each eye 912.The lens 910 includes an optical display component 914, such as a beamsplitter (e.g., a half-silvered mirror). The HMD device 902 includes anaugmented-reality emitter 920 that facilitates altering the brightnessof augmented-reality images. Amongst other components not shown, the HMDdevice also includes a processor 922, memory 924, interface 926, a bus928, and additional HMD components 930. The augmented-reality emitter920 emits light representing an augmented-reality image 940 exemplifiedby a light ray 942. Light from the real-world scene 950, such as a lightray 952, reaches the lens 910. Additional optics can be used to refocusthe augmented-reality image 940 so that it appears to originate fromseveral feet away from the eye 912 rather than one inch away, where thedisplay component 914 actually is. The memory 924 can containinstructions which are executed by the processor 922 to enable theaugmented-reality emitter 920 to perform functions as described. One ormore of the processors can be considered to be control circuits. Theaugmented-reality emitter communicates with the additional HMDcomponents 930 using the bus 928 and other suitable communication paths.The augmented-reality image 940 is reflected by the display component914 toward a user's eye, as exemplified by a light ray 916, so that theuser sees an image 918. In the image 918, a portion of the real-worldscene 950, such as, a countertop is visible along with the entireaugmented-reality image 940 such as a can. The user can therefore see amixed-reality image 918 in which the can is sitting atop the countertopin this example.

Other arrangements and elements (e.g., machines, interfaces, functions,orders, and groupings of functions, etc.) can be used in addition to orinstead of those shown, and some elements may be omitted altogether.Further, many of the elements described herein are functional entitiesthat may be implemented as discrete or distributed components or inconjunction with other components, and in any suitable combination andlocation. Various functions described herein as being performed by oneor more entities may be carried out by hardware, firmware, and/orsoftware. For instance, various functions may be carried out by aprocessor executing instructions stored in memory.

Having described embodiments of the present invention, an exemplaryoperating environment in which embodiments of the present invention maybe implemented is described below in order to provide a general contextfor various aspects of the present invention. Referring initially toFIG. 10 in particular, an exemplary operating environment forimplementing embodiments of the present invention is shown anddesignated generally as computing device 1000. Computing device 1000 isbut one example of a suitable computing environment and is not intendedto suggest any limitation as to the scope of use or functionality of theinvention. Neither should the computing device 1000 be interpreted ashaving any dependency or requirement relating to any one or combinationof components illustrated.

The invention may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program modules, being executed by a computer orother machine, such as a personal data assistant or another handhelddevice. Generally, program modules including routines, programs,objects, components, data structures, etc. refer to code that performparticular tasks or implement particular abstract data types. Theinvention may be practiced in a variety of system configurations,including hand-held devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The invention may alsobe practiced in distributed computing environments where tasks areperformed by remote-processing devices that are linked through acommunications network.

With reference to FIG. 10, computing device 1000 includes a bus 1010that directly or indirectly couples the following devices: memory 1012,one or more processors 1014, one or more presentation components 1016,input/output ports 1018, input/output components 1020, and anillustrative power supply 1022. Bus 1010 represents what may be one ormore busses (such as an address bus, data bus, or combination thereof).Although the various blocks of FIG. 10 are shown with lines for the sakeof clarity, in reality, delineating various components is not so clear,and metaphorically, the lines would more accurately be grey and fuzzy.For example, one may consider a presentation component such as a displaydevice to be an I/O component. Also, processors have memory. Werecognize that such is the nature of the art, and reiterate that thediagram of FIG. 10 is merely illustrative of an exemplary computingdevice that can be used in connection with one or more embodiments ofthe present invention. Distinction is not made between such categoriesas “workstation,” “server,” “laptop,” “hand-held device,” etc., as allare contemplated within the scope of FIG. 10 and reference to “computingdevice.”

Computing device 1000 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 1000 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media.

Computer storage media include volatile and nonvolatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer-readable instructions, data structures,program modules or other data. Computer storage media includes, but isnot limited to, RAM, ROM, EEPROM, flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other optical diskstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or any other medium which can be used tostore the desired information and which can be accessed by computingdevice 1000. Computer storage media excludes signals per se.

Communication media typically embodies computer-readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and another wireless media. Combinations of anyof the above should also be included within the scope ofcomputer-readable media.

Memory 1012 includes computer storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, non-removable,or a combination thereof. Exemplary hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing device 1000includes one or more processors that read data from various entitiessuch as memory 1012 or I/O components 1020. Presentation component(s)1016 present data indications to a user or another device. Exemplarypresentation components include a display device, speaker, printingcomponent, vibrating component, etc.

I/O ports 1018 allow computing device 1000 to be logically coupled toother devices including I/O components 1020, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, etc.

Embodiments described in the paragraphs below may be combined with oneor more of the specifically described alternatives. In particular, anembodiment that is claimed may contain a reference, in the alternative,to more than one other embodiment. The embodiment that is claimed mayspecify a further limitation of the subject matter claimed.

Accordingly, in one embodiment described herein, a method for enablinguser-interactions with virtual objects (VOs). The method may includeestablishing, by a head-mounted display (HMD) device, a communicationsession with an interaction-sensing (IS) device. The HMD devicedisplays, to a user, a field of view (FOV) that includes the virtualizedobject. The IS device is separate from the HMD device and is configuredto detect user-interactions including a user extremity position relativeto the IS device. The method may further include receiving from the ISdevice, via the established communication session, interaction datagenerated in response to a detected user extremity position relative tothe IS device. Additionally, the method can include modifying thevirtualized object included in the FOV based on the received interactiondata.

The IS device may be a touch and hover (TAH) device. The TAH device isconfigured to detect a touch of the user extremity on a first surface ofthe TAH device. The TAH device is also configured to detect a distancebetween the first surface and the user extremity when the user extremityis displaced from the first surface. In other embodiments, the IS deviceis a hover-sensing (HS) device. In some embodiments, the IS deviceincludes a touch-sensitive device, such as but not limited to atouch-sensitive display device.

The device may be further configured to mechanically couple with anoverlay. When coupled, the overlay presents at least a portion of asecond surface displaced from the first surface of the IS device. The ISdevice further generates the interaction data in response to a userextremity touch detected on the second surface. In at least oneembodiment, the IS device and/or the HMD device is configured toautomatically determine an identifier associated with the mechanicallycoupleable overlay for the IS device. The method may update an operatingmode of the HMD device, based on the identifier. For instance, theidentifier may be encoded in a portion of the interaction data providedto the HMD device.

The IS device may be configured to generate the interaction data inresponse to a motion of the user extremity detected along one or moresurfaces of a physical object. For instance, the object may be aprotrusion or shape included in the overlay. In other embodiments, thephysical object may be another physical object that is not part of theover. A shape of the physical object, such as the protrusion or anotherdiscrete physical object, may be automatically determined (by either theIS device and/or the HMD device) based on interaction data generated inresponse to the motion of the user extremity detected along one or moresurfaces of a physical object. The HMD device may update the FOV basedon the determined shape. For instance, a hologram or other VO may begenerated within the FOV that depicts a 2D or 3D visualization of theshape.

In some embodiments, the HMD device may update a rotational orientationof a VO included in the FOV based on interaction data that encodesdetected motion of the user extremity along one or more surfaces of aphysical object, such as but not limited to a protrusion of the overlay,or another separate physical object. The IS device may include multiplecamera devices configured to detect at least a portion of theuser-interaction.

The interaction data may encode a user-selection of a planar slice (orlayer) of a VO. In such embodiments, the method further includescorrelating, by the HMD device, the planar slice of the VO with atouch-sensitive surface of the IS device. The IS device is furtherconfigured to generate additional interaction data in response to a userextremity touch detected on the touch-sensitive surface. The HMD devicereceives, via the communication session, the generated additionalinteraction data communicated from the IS device. The HMD devicemodifies, the planar slice of the virtualized object included in the FOVbased on the received additional interaction data.

In various embodiments, the IS device is configured to generate theinteraction data in response to a motion of the user extremity detectedalong one or more surfaces of a protrusion included on an overlay thatis mechanically coupled to the IS device. The HMD device is enabled toupdate a position of the virtualized object within in the FOV based onthe received interaction data that encodes the motion of the userextremity.

In some embodiments, the IS device is configured to provide the userwith haptic feedback in accordance with an event within the FOV. Forinstance, in gaming applications, the a haptic-feedback interfaceincluded in the IS device may provide the user with haptic feedback.

In at least one embodiment, the IS device includes a two-dimensional(2D) capacitive-sensing surface. For instance, the IS device may includea 2D touch-sensitive surface such as but not limited to atouch-sensitive display device. Such IS devices are configured tointerface with an overlay that includes a three-dimensional (3D)surface. For instance, the 3D surface may be a protrusion. The overlaymay also include a plurality of capacitive couplers that capacitivelycouple portions of the 3D surface and portions of the 2Dcapacitive-sensing surface when the IS device interfaces with theoverlay. A one-to-one map between the capacitively coupled portions ofthe 3D surface and the portions of the 2D capacitive-sensing surface maybe generated. The IS device generates interaction data in response to auser extremity touch detected on the 3D surface. The HMD device updatesa 3D aspect of the virtualized object based on the received interactiondata and the generated one-to-one map.

The IS device or the HMD device may automatically determine a 3D contourof the protrusion or 3D surface of the overlay based on interactiondata. The interaction data may be generated in response to the usertouching or gliding their fingers along the 3D surfaces. The HMD devicemay generate another VO for display within the FOV. The shape of the VOis based on the determined 3D contour of the protrusion.

In another embodiment described herein, one or more computer storagemedia having computer-executable instructions embodied thereon that,when executed, by one or more processors, causing the one or moreprocessors to execute operations for enabling user-interactions withvirtual objects rendered in an immersive environment, is provided. Forinstance, the storage media may include non-transitory media. Theoperations include establishing, by a touch and hover (TAH) device, acommunication session with a head-mounted display (HMD) device. The HMDdevice is configured to display, to a user, a field of view (FOV) thatincludes the virtualized object.

The operations may further include generating, by the TAH device,interaction data in response to three-dimensional (3D) motion of a userextremity detected by the TAH device. The generated interaction data maybe communicated to the HMD device, via the established communicationsession. The HMD device is configured to modify the VO included in theFOV based on the communicated interaction data.

In some embodiments, the operations may further include receiving, via amechanical coupling, an overlay for the TAH device. The overlay mayinclude a protrusion. The TAH device is enabled to generate theinteraction data in response to 3D motion of the user extremity detectedalong one or more surfaces of the overlay protrusion. The overlayprotrusion includes at least one of a curved boss, a parallelepiped, acylinder, or a pyramid.

In some embodiments, the HMD is configured to update a viewpoint of theFOV or a position of the VO based on a portion of the communicatedinteraction data that encodes the 3D motion of the user extremitydetected along one or more surfaces of a protrusion coupled to the TAHdevice.

In various embodiments, the operations further include determining, bythe TAH device, a shape of a protrusion coupled thereto. Thedetermination of the shape is based on a portion of the generatedinteraction data that encodes at least a portion of the 3D motion of theuser extremity detected along one or more surfaces of the protrusion.The HMD device may be configured to update the FOV based on thedetermined shape of the protrusion.

In yet another embodiment described herein, a system for enablinguser-interactions with a VO in an immersive environment is provided. Thesystem includes a physical overlay. The physical overlay presents athree-dimensional (3D) protrusion. The physical overlay is coupleable toan interaction-sensing (IS) device. The 3D protrusion is displaced froma two-dimensional (2D) interaction-sensing surface of the IS device. Forinstance, the 2D interaction-sensing surface may be a touch- orcapacitive-sensing surface commonly employed in mobile devices such asbut not limited to smartphones and tablets. The physical overlay iscoupled to the IS device. The IS device is configured to generateinteraction data in response to at least a user extremity touch detectedon the 3D protrusion when coupled to the physical overlay. Ahead-mounted display (HMD) device is in communication with the ISdevice. The HMD device may be configured to update a field of view (FOV)displaying the virtualized object based on at least a portion of thegenerated interaction data received from the IS device.

In some embodiments, the IS device may be a touch and hover (TAH)device. The TAH device is configured to generate interaction data infurther response to a user extremity hover-gesture detected over aportion of the 2D interaction-sensing surface. The physical overlayincludes a plurality of capacitive couplers that are capacitivelycoupleable to portions of the 2D interaction-sensing surface when thephysical overlay is coupled to the IS device. At least one of the ISdevice and/or the HMD device is configured to generate a one-to-one mapbetween portions of the physical overlay and portions of the 2Dinteraction-sensing surface. The HMD device is further configured toupdate a 3D aspect of the virtualized object based on the receivedinteraction data and the generated one-to-one map.

The subject matter of embodiments of the invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

For purposes of this disclosure, the word “including” has the same broadmeaning as the word “comprising,” and the word “accessing” comprises“receiving,” “referencing,” or “retrieving.” In addition, words such as“a” and “an,” unless otherwise indicated to the contrary, include theplural as well as the singular. Thus, for example, the constraint of “afeature” is satisfied where one or more features are present. Also, theterm “or” includes the conjunctive, the disjunctive, and both (a or bthus includes either a or b, as well as a and b).

For purposes of a detailed discussion above, embodiments of the presentinvention are described with reference to a head-mounted display unit;however, the head-mounted display unit depicted herein is merelyexemplary. Components can be configured for performing novel aspects ofembodiments, where configured for comprises programmed to performparticular tasks or implement particular abstract data types using code.Further, while embodiments of the present invention may generally referto the head-mounted display unit and the schematics described herein, itis understood that the techniques described may be extended to otherimplementation contexts.

Embodiments of the present invention have been described in relation toparticular embodiments which are intended in all respects to beillustrative rather than restrictive. Alternative embodiments willbecome apparent to those of ordinary skill in the art to which thepresent invention pertains without departing from its scope.

From the foregoing, it will be seen that this invention in one welladapted to attain all the ends and objects hereinabove set forthtogether with other advantages which are obvious and which are inherentto the structure.

It will be understood that certain features and sub-combinations are ofutility and may be employed without reference to other features orsub-combinations. This is contemplated by and is within the scope of theclaims.

What is claimed is:
 1. A computer-implemented method for enablinguser-interactions with a virtualized object, the method comprising:establishing, by a head-mounted display (HMD) device, a communicationsession with an interaction-sensing (IS) device, wherein the HMD devicedisplays, to a user, a field of view (FOV) that includes the virtualizedobject, and wherein the IS device is separate from the HMD device and isconfigured to detect user-interactions including a user extremityposition relative to the IS device; receiving from the IS device, viathe established communication session, interaction data generated inresponse to a detected motion of the user extremity along one or moresurfaces of a physical object, wherein the motion is relative to the ISdevice; modifying the virtualized object included in the FOV based on adetermined shape of the physical object, wherein the shape is determinedby the IS device or the HMD device based on at least a portion of thereceived interaction data that encodes the detected motion of the userextremity.
 2. The computer-implemented method of claim 1, wherein the ISdevice is a touch and hover (TAH) device that is configured to detect atouch of the user extremity on a first surface of the TAH device, andfurther detect a distance between the first surface and the userextremity when the user extremity is displaced from the first surface.3. The computer-implemented method of claim 2, wherein the IS device isfurther configured to mechanically couple with an overlay that presentsat least a portion of a second surface that is included in the one ormore surfaces of the physical object and displaced from the firstsurface of the IS device, and generate the interaction data in responseto a user extremity touch detected on the second surface.
 4. Thecomputer-implemented method of claim 1, wherein the IS device is furtherconfigured to determine an identifier associated with a mechanicallycoupleable overlay for the IS device that includes the physical object;and updating, by the HMD device, an operating mode thereof based on thedetermined identifier of the overlay.
 5. The computer-implemented methodof claim 1, wherein the IS device is further configured to provide theuser with haptic feedback in accordance with an event within the FOV. 6.The computer-implemented method of claim 1, wherein the interaction dataencodes the detected motion of the user extremity along one or moresurfaces of the physical object, and the method further comprises:updating, by the HMD device, a rotational orientation of the virtualizedobject included in the FOV based on at least a portion of the receivedinteraction data that encodes the detected motion of the user extremity.7. The computer-implemented method of claim 1, wherein the IS deviceincludes a plurality of camera devices configured to detect at least aportion of the user-interactions.
 8. The computer-implemented method ofclaim 1, wherein the generated interaction data encodes a selection of aplanar slice of the virtualized object; and correlating, by the HMDdevice, the planar slice of the virtualized object with atouch-sensitive surface of the IS device, wherein the IS device isfurther configured to generate additional interaction data in responseto a user extremity touch detected on the touch-sensitive surface;receiving by the HMD device, via the communication session, thegenerated additional interaction data communicated from the IS device;and modifying, by the HMD device, the planar slice of the virtualizedobject included in the FOV based on the received additional interactiondata.
 9. The computer-implemented method of claim 1, wherein theinteraction data encodes the detected motion of the user extremity alongone or more surfaces of the physical object, which includes a protrusionincluded on an overlay, wherein the overlay is mechanically coupled tothe IS device; and updating, by the HMD device, a position of thevirtualized object within in the FOV based on at least a portion of thereceived interaction data that encodes the motion of the user extremity.10. The computer-implemented method of claim 1, wherein the IS deviceincludes a two-dimensional (2D) capacitive-sensing surface, and whereinthe IS device is further configured to: interface with an overlay thatincludes the one or more surfaces of the physical object and a pluralityof capacitive couplers that capacitively couple portions of the one ormore surfaces and portions of the 2D capacitive-sensing surface when theIS device interfaces with the overlay, generate a one-to-one map betweenthe capacitively coupled portions of the the one or more surfaces andthe portions of the 2D capacitive-sensing surface; and updating, by theHMD device, a 3D aspect of the virtualized object based on the receivedinteraction data and the generated one-to-one map.
 11. Thecomputer-implemented method of claim 1, wherein the IS device isconfigured to mechanically couple with a 3D protrusion that includes thephysical object, and further determine a 3D contour of the protrusionbased on the shape of the physical object; and the method furthercomprises: generating, by the HMD device, another virtualized objectedfor display within the FOV, wherein a shape of the other virtualizedobject is based on the determined 3D contour of the protrusion.
 12. Oneor more non-transitory computer storage media having computer-executableinstructions embodied thereon that, when executed by one or moreprocessors, causes the one or more processors to perform a method forenabling user-interactions with a virtualized object, the methodcomprising: establishing, by a touch and hover (TAH) device, acommunication session with a head-mounted display (HMD) device, whereinthe HMD device is configured to display, to a user, a field of view(FOV) that includes the virtualized object and a protrusion is coupledto the TAH device; generating, by the TAH device, interaction data inresponse to three-dimensional (3D) motion, detected by the TAH device,of a user extremity along one or more surfaces of the protrusion;communicating to the HMD device, via the established communicationsession, the generated interaction data, wherein the HMD device isfurther configured to modify the virtualized object included in the FOVbased on determining, by the TAH device, a shape of the protrusion basedon at least portion of the communicated interaction data that encodesthe 3D motion of the user extremity along the one or more surfaces ofthe protrusion.
 13. The one or more computer storage media of claim 12,the method further comprising: receiving, via a mechanical coupling, anoverlay for the TAH device, wherein the overlay includes the protrusion.14. The one or more computer storage media of claim 13, wherein theoverlay protrusion includes at least one of a curved boss, aparallelepiped, a cylinder, or a pyramid.
 15. The one or more computerstorage media of claim 12, wherein the HMD is further configured toupdate at least one of a viewpoint of the FOV or a position of thevirtualized object based on at least the portion of the communicatedinteraction data that encodes the 3D motion of the user extremity alongthe one or more surfaces of the protrusion that is coupled to the TAHdevice.
 16. A system for enabling user-interactions with a virtualizedobject, the system comprising: a physical overlay, aninteraction-sensing (IS) device, and a head-mounted device (HMD),wherein the physical overlay presents at least one three-dimensional(3D) protrusion, and is coupleable to the IS device, wherein the atleast one 3D protrusion is displaced from a two-dimensional (2D)interaction-sensing surface of the IS device when the physical overlayis coupled to the IS device, the IS device is configured to generateinteraction data in response to at least a user extremity touch detectedon the at least one 3D protrusion when coupled to the physical overlay,the HMD is in communication with the IS device and is configured toupdate a field of view (FOV) displaying the virtualized object based ona determination of a contour of the 3D protrusion that is based on atleast a portion of the generated interaction data received from the ISdevice.
 17. The system of claim 16, wherein the IS device is a touch andhover (TAH) device that is configured to generate interaction data infurther response to a user extremity hover-gesture detected over aportion of the 2D interaction-sensing surface.
 18. The system of claim16, wherein the physical overlay includes a plurality of capacitivecouplers that are capacitively coupleable to portions of the 2Dinteraction-sensing surface when the physical overlay is coupled to theIS device, wherein the IS device is further configured to generate aone-to-one map between portions of the physical overlay and portions ofthe 2D interaction-sensing surface, and wherein the HMD device isfurther configured to update a 3D aspect of the virtualized object basedon the received interaction data and the generated one-to-one map.